Skip to content

feat(worktree): add ScorpioFS backend - #447

Open
Ivanbeethoven wants to merge 9 commits into
libra-tools:mainfrom
Ivanbeethoven:feature/scorpiofs-worktree
Open

feat(worktree): add ScorpioFS backend#447
Ivanbeethoven wants to merge 9 commits into
libra-tools:mainfrom
Ivanbeethoven:feature/scorpiofs-worktree

Conversation

@Ivanbeethoven

Copy link
Copy Markdown
Collaborator

Summary

  • add backend-neutral worktree substrate and ScorpioFS adapter
  • preserve Libra ownership of Git state while ScorpioFS provides the FUSE data plane
  • document lifecycle, state reconciliation, Mega single-commit push behavior, and the validated boundary
  • include a regression test for clean detach with FUSE-local artifacts

Validation

  • ScorpioFS backend unit tests
  • worktree command integration test
  • default and worktree-fuse builds
  • deployed Mega project/aardvark-dns E2E: attach, status, fetch, add, commit, push --dry-run, detach

Ivanbeethoven and others added 7 commits July 22, 2026 13:50
Resolve 14 conflicts across 8 files and re-fit the ScorpioFS backend onto
main's reworked worktree registry.

Conflict resolutions, all keeping BOTH sides:

- src/internal/mod.rs, docs/commands/worktree.md, src/utils/util.rs and
  tests/command/worktree_test.rs: additive, each side's new item kept.
- src/command/worktree.rs and worktree-fuse.rs: this branch's `Scorpiofs`
  match arm alongside main's reshaped `List { schema_version }` and new
  `Doctor` arms; import block and `--help` EXAMPLES merged.
- src/command/status_untracked.rs: the candidate-path fast path now runs
  inside main's io-blocked collection and racily-clean guard —
  `collect_tracked_worktree_changes` takes the filtered `tracked_files`,
  and `scan_candidate_paths` gained the `io_blocked` field. It calls
  `scan_file`, which already records those events, so a blocked candidate
  propagates instead of vanishing.
- Cargo.lock: taken from main and re-resolved.

Fixes the merged tree needed to build:

- `scorpiofs = "=0.3.1"` was never published to crates.io, so the lockfile
  could not resolve at all. Repinned to `=0.4.0`, which exposes every API
  this branch uses — `cli::antares_overrides` (identical signature),
  `util::config::init_config_with`, `daemon::antares::{AntaresDaemon,
  AntaresServiceImpl}` and `new_external_state`.
- main reworked the registry: `WorktreeState::worktrees` is now `entries`,
  `WorktreeEntry` carries `worktree_id`/`state`/`epoch`, and
  `gc_worktree_scoped_rows` became `gc_worktree_scoped_rows_strict(db, id,
  directory_gone) -> Result`. The ScorpioFS attach/detach/rollback paths
  follow the patterns `worktree add`/`remove` already use: attach takes a
  `next_epoch()` registration and records `linked_history = Existed`,
  detach sweeps strictly (directory gone) and fails the command on error,
  and the attach rollback sweeps best-effort with a warning.
- `command_scope`'s exhaustive match had no arm for the new hidden
  `scorpiofs-worker`. Classified `Repository` like `libra service` rather
  than under-claiming it read-only, and added to the long-running
  exemption in `command_holds_shared_maintenance_lock` — a resident FUSE
  worker holding the shared maintenance lock for the life of the mount
  would starve every deletion phase.
- COMPATIBILITY.md and docs/development/commands/README.md rows plus a new
  docs/development/commands/scorpiofs-worker.md, required by the
  compat_matrix_alignment guards.
- Seven clippy findings under `-D warnings`: three collapsible_if, one
  needless_borrow, one field_reassign_with_default, and an OpenOptions
  `create` without an explicit `truncate`.

Verified: cargo check --all-targets --all-features, clippy -D warnings,
cargo +nightly fmt --all --check, and the compat_matrix_alignment suite
(8/8) all pass. The full `cargo test --all` run was still in progress at
commit time; CI covers it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ScorpioFS backend creates `<storage>/scorpiofs`, which the
`every_storage_rooted_file_is_a_classified_gc_source` guard requires to
appear in `GC_OBJECT_FILE_SOURCE_INVENTORY` or `NOT_AN_OBJECT_SOURCE`.

Register it as `NonRoot` rather than excluding it: `base_oid` names a
REMOTE monorepo revision the mount projects, not a local object-store
id, so rooting it would demand objects this repository never wrote --
the same shape as MERGE_RR. Libra owns the durable desired state here
precisely because the embedded worker persists none; the mount's own
Git writes reach the object store through the ordinary commit path,
which roots them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Unrelated to ScorpioFS, but `compat-offline-core` cannot go green
without them. Attribution: run 31259005383 (branch
codex/fix-windows-release-build, same day, no shared files) fails on
the byte-identical set of tests, so none of these are caused by this
PR. They are main's own, introduced after the last green run on
2026-07-27.

- worktree_scope (0ce8f77): five tests call `std::env::set_current_dir`
  directly. `#[serial]` only orders them against other `#[serial]`
  tests, not against the CWD_LOCK every `ChangeDirGuard` holds, so they
  yanked the process cwd out from under whatever fixture was mid-flight.
  Take the lock. `a_pinned_scope_survives_a_cwd_change` additionally
  pinned the ambient cwd, which `RequestScope::resolve` refuses outside
  a repository -- it only ever passed by stealing a moment when another
  test had parked the cwd in someone else's fixture, which is the same
  moment it broke that test. Give it a real repository.

- ai/hooks/runtime (6da73c6): `CaptureScope` entered the ingest path
  and resolves `RepoIdentity`, but `ingest_fresh_conn` never wrote
  `libra.repoid`, so 14 tests failed deterministically. Seed it, as
  `libra init` does. `ingest_fails_loud_when_table_missing` now drops
  one table from a complete schema instead of using a schema-less
  database, so it still tests what it claims; the tombstone test asserts
  against the claim diagnostic that now carries the refusal.

- utils/path: the commondir fixture's target lacked the repository
  database, so it is no longer TERMINAL common storage.

- rename_detect (5c1b7bb, a26cc9a): the LFS classification resolves
  `working_dir()` infallibly on the pooled io thread, which panicked the
  worker outside a repository and surfaced to the caller as `IoTimeout`.
  Give both tests a repository. A jammed pool reports `IoTimeout` too,
  and the pool is process-global -- the seam tests abandon reads that
  keep sleeping in a worker past their own end -- so the negative test
  drains the pool and retries. A seam that really fired would still blow
  every attempt.

Verified: `cargo test --lib` 4403 passed, 0 failed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@genedna

genedna commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

CI 归因说明(为什么这个 PR 里有几处与 ScorpioFS 无关的测试改动)

compat-offline-core 上共 19 条失败,其中只有 1 条是本 PR 造成的:

  • command::maintenance::tests::every_storage_rooted_file_is_a_classified_gc_source —— ScorpioFS 后端新建了 <storage>/scorpiofs,未登记进 GC 来源清单。已按 NonRoot 登记(base_oid 是远端 monorepo 的 revision,不是本地对象库 id,与 MERGE_RR 同形态)。

其余 18 条是 main 的既有红线。对照实验:同一天(2026-08-08)的 run 31259005383,分支 codex/fix-windows-release-build,与本 PR 无任何共同改动文件,失败列表与这里逐字相同。它们都出现在 2026-07-27 最后一次绿灯之后:

来源 commit 影响 根因
0ce8f77c (07-31) 2 条 worktree_scope + 连带 5 个测试直接 std::env::set_current_dir,未持有 ChangeDirGuardCWD_LOCK#[serial] 与之是两套锁。另外 a_pinned_scope_survives_a_cwd_change pin 的是进程 cwd,而 RequestScope::resolve 在非仓库目录下不装 pin
6da73c67 (08-05) 14 条 ai::hooks::runtime CaptureScope 接入 ingest 路径后要解析 RepoIdentity,但夹具 ingest_fresh_conn 从未写入 libra.repoid(确定性失败,跑单条也挂)
5c1b7bb2 (07-28) / a26cc9ac (08-05) 2 条 rename_detect LFS 分类在 io 线程上调用不可失败的 working_dir(),非仓库目录下直接 panic worker,调用方只看到 IoTimeout
1 条 utils::path commondir 夹具的 target 缺少仓库数据库,已不再满足 TERMINAL common storage

修复都收在 a05d3491 一个 commit 里,与功能改动 9a4736fa 分开,方便单独 review 或摘出。

本地验证:cargo test --lib 4403 passed / 0 failed;cargo +nightly fmt --all --checkcargo clippy --all-targets --all-features 均干净。

genedna and others added 2 commits August 9, 2026 12:19
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`compat-offline-core` hit GitHub's 360-minute cap. It was not a
slowdown: the lib suite ran in 2150s, the same as the 2125s it took on
the last baseline. What changed is that the lib suite stopped failing,
so `cargo test --all` no longer aborted there and reached the rest of
the suite for the first time in weeks.

The cost is concentrated in one target. `tests/command_test.rs`
compiles 150 modules into a single binary -- deliberately, to avoid
paying for one binary per command -- and ~1130 of its 2956 tests take
the process-global cwd lock that `ChangeDirGuard` holds. It is
therefore lock-bound, not CPU-bound: 479 of its tests were logged as
running for over 60 seconds while the machine sat at ~0.3 load. Adding
cores does nothing; only another PROCESS does.

So split it out into `compat-offline-command`, a 2-shard matrix. Each
shard is its own job, hence its own process and its own cwd lock, and
the split scales with the runner pool rather than the core count.

Coverage is identical to `cargo test --all`: lib, bins, doctests and
the other 208 integration targets run in compat-offline-core, and
command_test runs in the shards. The core job enumerates its targets
from `cargo metadata` rather than a hand-written list, and both jobs
refuse to run a suspiciously small set instead of silently passing on
partial coverage.

Shards partition by enumerated test NAME, not by module prefix: the
modulo split is exhaustive and disjoint by construction, so a rename or
a new test cannot fall out of every shard. Verified locally: 2956
tests, 1478 per shard, union 2956, intersection empty.

Both jobs also get `timeout-minutes: 350` so an overrun fails as a
timeout with its log intact rather than being reaped at the cap.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants